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ABSIRACT ^ , . ' ^ ^ 

Effective instruction on a naticnal Scale is 
dependent on supplyin'g competent teacher'^ with comprehensive 
instructional materials and procedures that meet some kind of minimum 
criteria for accountability. Considerations included in a complete 
instructional system are: , (1)-' outcomes; (2) assessment; (3) user 
experience data; (U) materials; (5) training; (6) reporting; (7) time 
and costs. The prograini* -musi specify outcomes stating what the ; 
learners will he able to do as a result oi instructicn. Whet* the 
outcomes are clear;Ly stated, educators can validly as'sess whether 
prog"ram is potentially worthwhile * and appropriate for a pa^iculai:' 
population of s,tudents.' Measures should be included that fre-guently 
assess pupil progress, and assessment materials should be directly ^ 
keyed tp a clearly defined set of - outcomes.* Empirical data^ regarding. ' 
the devdlcpaent of an instructioncil program must not be overlooked. 
Data mu^ be presented to indicate that outcomes hav^ been 
consistently attaine^'d during . previous use of the ^riigram. in a wide" 
range of situatiois 'eaid that vlearners a'nd te^chers^ have expressed 
-satisfaction with the system,!, Included must be ins'fructionar and 
supplemental materials and actiyi'^es that are keyed directly to t^i^ 
expe'cted program gutpcmes and^assessment materials. Materials for 
training teachers and other school personnel and for monitoring 
program performance ar^ necessary. Time and cost effectiveness are 
the final cocsid^ra tiCTis. (DMT) . ' , . ^ 
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ABSTRACT 



'Characteristics of "a complete ins;tructio*nal' sySte^rr are ^ef iT?ed-.^ 

• ' \ ^ 

Considerations inclu(Jed ar§ outcomesV assessment ; nsei' experieii'ce ' 
data, materials, training, reporting,, and tdme ^nd tosts;' Eacii 
consideration is elaborated to ipdiCate how it; c(Jnttibutes^ to:tlie J^t 
effective operation of an instructional 'systfem. . T 
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-CONSIDERATIONS IN' SELECTING INSTRUCTIONAL TROCIUNS 
Fred C. Niedermeyer and Michael H. Moncri|f , * 

.Selection of instructional progran^s and materials for schools is an 
extremely important task. But .on what basW^ ,do school ayithorities decide 
which reading series or which history, text ,to purchase,, and use in' their^ - 
classrooms? Because complete instructional programs, that contain an 
adequate accountability base for teaching irapo^'tant learner outcomes 
have not existed in the pa5t and are in short supply at the present, 
schools have had -to select textl>ooks and supplementary resource 
materials lacking in analytical and emp'iriqal soundness*: Primarily, 
school authoriti.es, have based their selection deci3ions on analyses 
of the content: and, in good measure, on the cosmetics of the materials 
themselves. As a- cOnsequeVice , with the great variability in teacher ^ 
competence and pr^actice ,' schools are often disappointed with the 
resulting record , of', pupil performance. 

* ' In recent* years, . publishers, educators, and the-.R & D community 

.have begun to addtess,, Che problem. Uniformly, they realize that 

^demonstrably "more effective insti'uction on a national scale is dependent 

upon supplying comjie^tent teachers with comprehensive instructional 

materiai^s and procedures that meet some kind of minimum criteria with 

respect to ^ajnalytic and empirical s'oundness. > '* . - 

Hereid lies ''the problem: How dc5es the concerned educator see to it 

that quality iftaterials .are produced that do indeed meet minimum criteria? 
" . ' , \ ' ^ 

He doesn^ t',, directly . But as ' a ''prof essional decision-raaker , the consi- 

* ' ' ' ' ' 

derations he uses in selecting .Mistruct^onal programs," if ' sufficiently 



uniforjri across the prof.ession, will quigkly servfe to establish indtis.try'/ 

standards. Not so quickly produced, however, are 'the resulting instruct 

tional' materials a^ procedures that reliably' produce prespeci£ied resuljts 

Patience from the prpfession n\ay be required, but not without cl continu^ 

ing^ hard line o'f q*uestioning. ' , • 

. *» '. , . ' * 

. ^ ' 

Figure 1 contains 'a checklist of questions which, if answered 
'positively, will go far in assuring sch90ls of'effectivfe instruction 
on a national scale. ' The' considerations reflected' in the* checklist . ' 
are not 'intended to be exhaustive; the individual user may wish to 
consider many other possible characteristiQS of an instructional pi:ogram. 
However, the considerations are intended to define a complete instruct 
tional program. A program that does not include thfese elements is . ' ' 
incomplete., and s^hould- be tecognized as such. 

CONSIDERATION 1: Does the instructional program specify outcomes ' ' 

stating what the learners will. -be able to do as a 
^ result oY *jLnstruction? > ' . 

11: is on'ly when the outcomes to be accomplished by the leathers in 

a program aire* clearly stated that educat9rs can validly assess whether 

a program is potentially worthwhile and appropriate f.pr a -particular 

population of. students. For example, one outcome of a beginning reading 

program might bre for children to. sound out new^words composed of 

previously learned le-ttef sounds. \ One outcome of a social studies 

program might be for students* Co distinguish fact from opinion in 

*hist"otical 'writings and news articles. Q^, an art ptogram might include 

an Qutcome in which studen^ts draw human and animal figures in correct 

size proportion. " ' . ' 
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• * ' • . . . Figure 1 . 

, Checklist 'of Considerations in. Selecting Instructional Programs 
— ^ : ^ -t ' 
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X.\ ' Outcomes .. t)oes the iustnrp'tional prograp speci'fy outcomes 
• ' fe^tating. wh^t the learuers w^ill "be dble to do'as a result of 
^ instruction-? , ' , * ' . * ' • 



2. Asses-siaenD , , Ar.e^ measures "included* that frequeatly assess pupil 
^ progress t<?wards program 6utc6paes?' ' ^ . ' 



D";.3l. User Experience Data' ,- ^ Are data presented tb indicate that 

(1) 'outcoi^es'^have been ct)nsistently 'attained during previous ^ 
Us? o'f the praflcram in a wid^ range^of situations an<J (2) learner^, 
tochers, and ^t'r\ex^ involved in th^ program YAve expressed 



satisfaction with th^ system? 

\ 7 ^ - . 



^"Cn ' Matei:ials . Are instructional aAd supplemental materials and ^ 
• activities iholuded tfiat.are keyed directly to the expected 
' program outcoraes'^nd assessment/materials? ^ 

O 5«' Training/ , .Are mater i^ls* and procedures included for training 
" . . teac'her§ and other school "personnel '-to use the specified instruc-^ 
; . tional *matferials and procedure^? ^ ' \ ' 

• •[!] 6.' ^epprCin^ , •Are.jnate::ials 'and procedures included ^for teachers ' 
'and administrators to monitor program-performance sc as to 
. . identify ^d coi;rect instructional problems? ' ' ^ 

' 1 1^ 7.\ Tinfe. and' Costs .. Ar'^ the time, and cost require'ments for establishing 
, . '^ind operating the.iustructional system acceptable to those involved * 
^ 'and Reasonable', in 'tferms pf expected outcomes for learners'? 
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Spetif'ic. diitcom^ statements should' not be confused with the global . ' ,/ ^ 
rhet^QTic ' that accprapanies many programs. For example, the statements ' ^ 

' * *iS*tudents will lea-rny^itizenship' -skills required for a democratic 

^ , ' **'V • "\ ' ■ * " * r . • ' ' ' ^ 

society" 'or '^^Chiidren will exhibit'^ lO-'month growth on a standardized 

reading test at the end of six m^htlVs" arfe^ vague arid tell little 'ab(5ut * ' * 

'wha't students will actually learn to dt> djari^g, the school year^ • . . 

• When if^struc tional qutcomes ar-^ well-stated, programs canl)e-'.- . ' 

compared on the basis of the skills to be taugTit. Educators, parents, * , ' . 

and community members can then determine which set of o.utcomes appears 

^ most worthwhile "and appropriate for students ^to ^ttain. Alrso,^ as will 

be 'S^en, only ^instructional programs based on <;:learly specified out- , 

comes will be ^b.le to satisfy. many t»f ^the remaining' cbnsid^^rations ; ' ; ' " 

CONSIDERATION ^2 : ' Are m^^es included that' frequently assess p^frij.., . ' ' . • " 

- . ' progres^Ct^wartis .proyam outcomes^ ' [ *Z>' . - \, ^ ^ '\ 

. ' -Without approplriate^as'sess^n^pt ma*terials, and. procedures , it is x 

'?near:ly ijnpq^ssible tq^determine if a, program i^, effectively following 

. through on.it>s stated outcdine3. AsseSstjent^ materials should be.direqtly 

keyed 'to, a^. clearly ^def ined, yet .of outcpmes . For ^xampje,, if an ou'tcpme - . 
States that-*"^tud.etlt^ aria 'to draw huii^n figures yi£h tj^e bpdy and-( facial ' - *' 
partS'. in correcr ' size'/prppbrtidn, .trien ,the 'a^se^sment shouj-d requite 
Students, to 4raw human'. fig.ures*^*-^not answer Xrue-*f^lse questions about 
' ' 'the; various "^fi^bportidns . ''^ ^ ' ' ^' t. ^ ' * . - ' ' * ' \ * ' 

. ^ Many ^diffdr exit, fdvms o& assesstnent are possible, including obser^t' ' , 
vatiions, writt^^h and^ 'dt'^J^'tifests , Vand corapl«et^d prpXects. Wliatever* form ^ 
o;^ jassQjs^mep.'t - i^.provided> however,^ it should ^specifically evaluate the ^ 
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stated outcomes of the program,* so as. to allow^ the teacher .tof ma-ke 

instructional decisions about ^Individual students. 

. ' .Many programs provide achievement^ test^ for us6 at the* end of the 

•SQh4oI year, and these are useful for sumniative. eydluation of the 

program^ However, more freqXient assessment is aiso.^ needed. To 'help 

a student learn-effectively, the teacher nefeds to kno\^\^xactly , where 

the student is^ha^vin^ difficulty. This information is lie^Sfd throughqut 

the^school year when there is still time to provide the* needed instruc^i 

Progriam§ that /cont^i^n appropriate assessment materials for a se- 

quence of outcomes over d range of sicill-^- levels lend thefDiselves. to the' 

multji-rlevgi or non-graded appraach\ 'Entry-skills tests permit the . • 

/pl4cejjfen(^-^f' children in these programs at level's correspQnding,tp th6ir 

^ §kills .or ^^iXitifes, rather 'than to their age, or grade in school... , 

^» ' ^ * ^ ' ' ^ • ! ' 

^ Qfiildren ;nay t^eq. progre^ through the program, at a level that Is . • 

jrealistiyt and apprppriate to their needs and abilities.'. - ^ 

Ay additipnal advantage^ of an outcomes-ba.sec}. prpgraiit containing 'I 

appfopriate assessment is that teachers and pupils find it i;ewarding 

and motivating ' to i^egularly se*e empirical evidence of growth. In a ' 

wel^^-developed instructional system, the results of assessment are 

usually; positive and they can be shared with those Concerned. If 4:he 

results are not positive, then it "signals that, changes need to .-be 

formulated and tried out. Subsequent assessments will evaluate the 

\ ^ i ^ . • 

/(^fficacy of such changes. 



CONSIDERATION 3: ^ Are data pt^sented to indicate tha^ (1) outcomes have 

' . * . been consistently attained diiring ; previous us6 of the 
y ' ' ^* program. in a' wide range of si tuations. and (2) learne/s, 
- -* " _ teachers, -and others involved in the prosraio have 'expressed 

• ' Satisfaction with the systeint 

The AnSjweie' to this question" determine^ whether a program has Actually 

'been quality verified in .schools or whether it is a . first-draft effort" of . 

.^undetermined e'f f ectiveness. Reports of development of outcoines-based 

♦ instructional system^ invariably show how early versions of ^ program 

are modified considerably to increase its ef f ec tiveness • Through, a series 

o'f revision's^^d tryouts, "bugs" can be worked out of -a program* until it 
, do^s reliably promote the intended outcomes, A program field tested in 

a variety of school locations and situatiohs allows- the potential user to. . 

bette?: estimate program , operation for hi<s' own situation. Yet, ir^ a 
-fairly recent, ETIE report . (American Education ^ 4971), il: was estimated • 

th^t less thap two percent of all inptriSctiopal .programs available on the 

educational market at that .-^time had been tested pcior to* publication/ 

Empirical daxa regarding^ the development of'' an instructional' program ^ 

must not be overlooked. 'Too often developers and publishers rely on, 

• • ■ ■' , ^ ' ■ • v ■ ■ 

few carefully- clio'sen testipionlais to sul^stantiaJte the credibility of a 

'*\ - ' ' ■ V - - . • •"' * "■ 

.programi .'I'/liiJ.e such testimonials^rq^ ^n^ariably positive, they^ may xtyt . • 
be representative of .thfe. majority of usek;s:. Quantitative data should be 
presented when Reporting the cogriitive apd affective effectiveness of a 
' prog^ram. " ^ ' . • * - •* ' . 

^> ■ ; ' ^ . . • • 

, P.rinlary;' criteria for determinjLng the effectiveness of; an Instructional 

* "* • * - ' . 

/program ar^ data^ resulting from assessment of the program's statpd putcomes. 

'/ * ' • * ' * 

Tfies^ data should clearly indicate the contribution of 'the instructional 
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prpgram^ih promoting the outcomes (i.e. , baseline da-ta and control groups) 

; . ' / - - « , ' . ' ' 

They should be straightforward 'and interpretable j 60 tha^ a s'tatisti- , 
ciati is. not' require'^d, to* e^yaluate their credibility-. 

CONSIffERATlON' 4 Ar^ instructional and supplemental materials and 

acti^j ^ it^es included that are keyed directly to ^tbe ' • 
expected ' program outcomes and assessment materials? " 
^-p^ . . * - ^ . . • 

-^ination of inst'ructional materials has traditionally ^been the 

primary basis for proj^rani selection. .Although factors such as ^visuai < 

appeal ^nd overall durability are important selection considerations, 

they should not' ou\weigh the instructional value of the materials 

themselves. 

Instructional materi^^ should be keyed directly to the outcome^' 

. - ■■ -; ■ 

of the program; - that is<, tHey should provide the means for the student 
to practice arid improve tH^ skills defined in the program's* putcomes . 
If the stated Outcomes of an art program include drawing, basic " shapes 
knd desig^ns, for example, the instructional matetials should provide 
children opportunities to practice these. skills. Of course, it is also 
imperative that instructional materials' be Interesting and appealing to' 
students, and' be convenierit and easy f.djr teachers to 'use. , ' ' 

In addition, iristructional materials should^ include procedures j>r 
suggestions fot supplementary activities for children whose; performance « 

in the program's asse.s^ment checks indicate the need for further instruc- 

* \ ' ' 

tion. "Supplemental practice can take a number of forms, :|.ncluding- 

specially designed exercises that are individually administered by \ 

student tutors or adult paraprof e'ssionals. Whatever form supplemental ^ 

instruction takes, it should be directly related to the program outcome^. 

\ . " 

1 , - <• . 

1 1 - 



- A 




CONSIDERATION 5: Are materials and procedures included*- for^^traininR 

teachers and\6ther school persoyinel to use\the speci- 
^ fied instructionar ^nd issessineut materials^ "^nd ^ 

procedures? ' ' , ^ ~' ~ 

The larger the p*rogram, the more import^^^iLt this consideration becomes. 

. For example, the avaitability of training raatBrials is, more critical 

.when selecting a complete elementary ' school reading program than' when 

seTecting a twd^week unit on map skills. . For outcomes-based programs 
* \ A, ■ * 

^of.at least a semester or year in duration, a traditional teacher 's text 
is usually insufficient for acquainting ^teachers with the procedures for 
conducting the program. The instructional, assessment, and supplemental- 
activities ^in an- outcomes^based program are nev. to most teachers and 

• ' ;^ ' ' ■ ^ " 

require the use of specialized materials and procedures (Niede^meyer , 1971). 

^'Mapy pi:ograms of fer 'excellent dnstf uctional manuals and training • These 

^J- ^ ' ' ' . ' 

. training materials clearly idetitify and define the responsibilities of 

,all 'personjiel^ itivolved in program operation, including supervisors, 

. ' ■ / * ^ - V * 

. ^(iminls^ratdr-^y pataj^rof essionals^, and par^nt^, as well as teachers. 

\ ^ /To hej.R ^teachefs .attain ins'tifuctional success, the training should , 

pfovi^^ cleaT'^stan^ards. or expettjartions of class performance. For ex- 

atiip-lfe, teachers shpuld be given some idea of the Average time required 

^for most class&s to complete -an instructional .unit, and of the I'evels 

of achievement that most children •shoul4 attain. * ^ ^ ^. 

'\ ^ : * . , . 

CONSID.ERATIO'N' 6 : Are materials an4 procedures included f6r^ teachers and 
' \ administrators ^ tb monitor program- peffo'i::mtoce so- as to 

/ ' ' s. ' identify and correct ins tructional ■ prog^am^?^ 

'*A complete instruetionail 'progtatp eont^itis. a system for reporting 
... * \ V ' ' • 

* ' . \. ' ^ ' 1- , ' . . 

cla§9» pi;o^ress on a reg.ular basa.^,.ilot only-'to ^help teachers and super- 

"vifeors impt.6ve pupil ;performan'ce, but 'also' to keep. releVant audiences 
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(parents, principals, teachers, supervisors, hoard 'members) informed 
of class, school, and district progres^^^luring the year. This performance 
reporting system should be keyed" to the outcpmes and assessment materials . 
defined for the instructional program. 

The performance reports can be completed by -the teacher, or a 
computerized system can be used to Score tests and print out status 
reports. In either ca^e, teachers and administrators will have avail- 
able report data to identify those areas where instruction needs to ^ 
be analyzed an^d mcfaifieS^. It is important that the system include tested 
procedures for teachers and supervisors to use when analyzing and modifyirtg 
li^truction to remediate/learning problems (Niedermeyer & pischer, 1974). 
* With a performance reporting system, the notion of ^instruction 



accountability becomes operational in a positive way. Parents are right- 
fully informed of their children^ s specific skill proficiencies. Teachers, 
administrators, and supervisors have the means to identify and correc't 
instructional .problem^ when and where they occur.- In short, on the* ^ 
basis of pupil performaace data, educators can make substantia]^ Z 
instructional decisions and' rid themselves of the ^bldme-casting mold , 
traditionally used td explain away classroom learning failures* 

CONSIDERATION 7: Are the time and cpst requirements for establishing 

and operating the instructional system acceptable- to 

^ those involved and r;easonable in terms of expected 

learner outcomes?, 
\ 

> Time ^nd costs involved the final cons^ideration in selecting an 

"r ' ' - • 1 ^ . ' ' ' • . , I ' 

instructional program. Some programs req^uire exte^feive teabher training » 
(e.g., up to six'weeks); others require the involvement .of mlfjky persons 
and perhaps a substantial curriculum adjustment. The ^school staff must 



he prepared to accept^ insta-11, and oj>erate^' sGch programs. The decision'. 

as to' whether the, time andiexpeose invQlved is worth the effort iftr terms 

of the benefitb to be der ived" '""fTom the prqgram ultimately 'involves the . 

nature and cuality of^ the instructional prosran and Its outcomes. If 

intX)inDlete or ^ poor ly.. conceived and d eve-lope J , the' program, is' likely to 

nake inefficient, use of everyone's time arud should not be selected. 

Clearly, the t>T)e of pro ^;raiTi .defined by^these considerations (Fig, 1) 

has requirements different from' traditional instructional texts. Systemati 

. * ' ' ^ \ ' • . ^ 

cally c'eve loped, research-based instpuctiona^l programs a're relatively new, 

and have not as yet appeared on the educational^markefe in large numbers., 

liaising the questions ^inheirent in t'he considexatdons, wiH\ however^ in- 

crease the likelihjj||)d 'that 'publisher s will accelerate development and 

availability of such programs. As th^se programs do become available, 

tho£>e interested in obtaining effective, reliable instruction for their 

i n 
schools will do well to seek them out* • ' 
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